Focusing on the target tracking problem in the docking stage of autonomous aerial refueling， a joint detection and tracking algorithm of target in aerial refueling scenes was proposed. In the algorithm， CenterTrack network with integrated detection and tracking was adopted to track the drogue. In view of the large computational cost and long training time， this network was improved from two aspects： model design and network optimization. Firstly， dilated convolution group was introduced into the tracker to make the network weight lighter without changing the size of the receptive field. At the same time， the convolutional layer of the output part was replaced with depthwise separable convolutional layer to reduce the network parameters and computational cost. Then， the network was further optimized to make it converge to a stable state faster by combining Stochastic Gradient Descent （SGD） method with Adaptive moment estimation （Adam） algorithm. Finally， videos of real-world aerial refueling scenes and simulations on the ground were made into dataset with the corresponding format for experimental verification. The training and testing were carried out on the self-built drogue dataset and MOT17 （Multiple Object Tracking 17） public dataset respectively， and the effectiveness of the proposed algorithm was verified. Compared to the original CenterTrack network， the improved network Tiny-CenterTrack reduces training time by about 48.6% and improves the real-time performance by 8.8%. Experimental results show that the improved network can effectively save the computing resources and improve the real-time performance to a certain extent without the loss of network performance.